{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 1.ERNIE 3.0 轻量级模型简介\n",
    "\n",
    "PaddleNLP 开源的 [ERNIE 3.0 轻量级模型](https://github.com/paddlepaddle/PaddleNLP/tree/develop/model_zoo/ernie-3.0) 是在文心大模型 ERNIE 3.0 基础上通过在线蒸馏技术得到的轻量级模型，模型结构与 ERNIE 2.0 保持一致，相比 ERNIE 2.0 具有更强的中文效果。\n",
    "\n",
    "相关技术详解可参考文章[《解析全球最大中文单体模型鹏城-百度·文心技术细节》](https://www.jiqizhixin.com/articles/2021-12-08-9)\n",
    "\n",
    "# 2.模型效果\n",
    "\n",
    "ERNIE 3.0 轻量级模型开源 **ERNIE 3.0 _Base_** 、**ERNIE 3.0 _Medium_** 、 **ERNIE 3.0 _Mini_** 、 **ERNIE 3.0 _Micro_** 、 **ERNIE 3.0 _Nano_** 五个模型：\n",
    "\n",
    "- [**ERNIE 3.0-_Base_**](https://bj.bcebos.com/paddlenlp/models/transformers/ernie_3.0/ernie_3.0_base_zh.pdparams) (_12-layer, 768-hidden, 12-heads_)\n",
    "- [**ERNIE 3.0-_Medium_**](https://bj.bcebos.com/paddlenlp/models/transformers/ernie_3.0/ernie_3.0_medium_zh.pdparams) (_6-layer, 768-hidden, 12-heads_)\n",
    "- [**ERNIE 3.0-_Mini_**](https://bj.bcebos.com/paddlenlp/models/transformers/ernie_3.0/ernie_3.0_mini_zh.pdparams) (_6-layer, 384-hidden, 12-heads_)\n",
    "- [**ERNIE 3.0-_Micro_**](https://bj.bcebos.com/paddlenlp/models/transformers/ernie_3.0/ernie_3.0_micro_zh.pdparams) (_4-layer, 384-hidden, 12-heads_)\n",
    "- [**ERNIE 3.0-_Nano_**](https://bj.bcebos.com/paddlenlp/models/transformers/ernie_3.0/ernie_3.0_nano_zh.pdparams) (_4-layer, 312-hidden, 12-heads_)\n",
    "\n",
    "\n",
    "下面是 PaddleNLP 中轻量级中文模型的**效果-时延图**。横坐标表示在 IFLYTEK 数据集 (最大序列长度设置为 128) 上测试的延迟（latency，单位：ms），纵坐标是 CLUE 10 个任务上的平均精度（包含文本分类、文本匹配、自然语言推理、代词消歧、阅读理解等任务），其中 CMRC2018 阅读理解任务的评价指标是 Exact Match(EM)，其他任务的评价指标均是 Accuracy。图中越靠**左上**的模型，精度和性能水平越高。\n",
    "\n",
    "图中模型名下方标注了模型的参数量，测试环境见[性能测试](https://github.com/paddlepaddle/PaddleNLP/tree/develop/model_zoo/ernie-3.0#%E6%80%A7%E8%83%BD%E6%B5%8B%E8%AF%95)。\n",
    "\n",
    "batch_size=32 时，CPU 下的效果-时延图（线程数 1 和 8）：\n",
    "\n",
    "<table>\n",
    "    <tr>\n",
    "        <td><a><img src=\"https://user-images.githubusercontent.com/26483581/175852121-2798b5c9-d122-4ac0-b4c8-da46b89b5512.png\"></a></td>\n",
    "        <td><a><img src=\"https://user-images.githubusercontent.com/26483581/175852129-bbe58835-8eec-45d5-a4a9-cc2cf9a3db6a.png\"></a></td>\n",
    "    </tr>\n",
    "</table>\n",
    "\n",
    "batch_size=1 时，CPU 下的效果-时延图（线程数 1 和 8）：\n",
    "\n",
    "<table>\n",
    "    <tr>\n",
    "        <td><a><img src=\"https://user-images.githubusercontent.com/26483581/175852106-658e18e7-705b-4f53-bad0-027281163ae3.png\"></a></td>\n",
    "        <td><a><img src=\"https://user-images.githubusercontent.com/26483581/175852112-4b89d675-7c95-4d75-84b6-db5a6ea95e2c.png\"></a></td>\n",
    "    </tr>\n",
    "</table>\n",
    "\n",
    "batch_size=32 和 1，预测精度为 FP16 时，GPU 下的效果-时延图：\n",
    "\n",
    "<table>\n",
    "    <tr>\n",
    "        <td><a><img src=\"https://user-images.githubusercontent.com/26483581/175854679-3247f42e-8716-4a36-b5c6-9ce4661b36c7.png\"></a></td>\n",
    "        <td><a><img src=\"https://user-images.githubusercontent.com/26483581/175854670-57878b34-c213-47ac-b620-aaaec082f435.png\"></a></td>\n",
    "    </tr>\n",
    "</table>\n",
    "\n",
    "从图上可看出，ERNIE 3.0 系列轻量级模型在精度和性能上的综合表现已全面领先于 UER-py、Huawei-Noah 以及 HFL 的中文模型。且当 batch_size=1、预测精度为 FP16 时，在 GPU 上宽且浅的模型的推理性能更有优势。\n",
    "\n",
    "在 CLUE **验证集**上评测指标如下表所示：\n",
    "\n",
    "<table style=\"width:100%;\" cellpadding=\"2\" cellspacing=\"0\" border=\"1\" bordercolor=\"#000000\">\n",
    "    <tbody>\n",
    "        <tr>\n",
    "            <td style=\"text-align:center;vertical-align:middle\">\n",
    "                <span style=\"font-size:18px;\">Arch</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px;\">Model</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px;\">AVG</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px;\">AFQMC</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center;\">\n",
    "                <span style=\"font-size:18px;\">TNEWS</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center;\">\n",
    "                <span style=\"font-size:18px;\">IFLYTEK</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center;\">\n",
    "                <span style=\"font-size:18px;\">CMNLI</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center;\">\n",
    "                <span style=\"font-size:18px;\">OCNLI</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center;\">\n",
    "                <span style=\"font-size:18px;\">CLUEWSC2020</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center;\">\n",
    "                <span style=\"font-size:18px;\">CSL</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center;\">\n",
    "                <span style=\"font-size:18px;\">CMRC2018</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center;\">\n",
    "                <span style=\"font-size:18px;\">CHID</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center;\">\n",
    "                <span style=\"font-size:18px;\">C<sup>3</sup></span>\n",
    "            </td>\n",
    "        </tr>\n",
    "        <tr>\n",
    "            <td rowspan=3 align=center> 24L1024H </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">ERNIE 1.0-Large-cw</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\"><b>79.03</b></span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">75.97</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">59.65</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\"><b>62.91</b></span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\"><b>85.09</b></span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\"><b>81.73</b></span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\"><b>93.09</b></span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\"><b>84.53</b></span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\"><b>74.22/91.88</b></span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\"><b>88.57</b></span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\"><b>84.54</b></span>\n",
    "            </td>\n",
    "        </tr>\n",
    "        <tr>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">ERNIE 2.0-Large-zh</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">76.90</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\"><b>76.23</b></span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\"><b>59.33</b></span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">61.91</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">83.85</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">79.93</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">89.82</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">83.23</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">70.95/90.31</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">86.78</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">78.12</span>\n",
    "            </td>\n",
    "        </tr>\n",
    "        <tr>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">RoBERTa-wwm-ext-large</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">76.61</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">76.00</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">59.33</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">62.02</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">83.88</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">78.81</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">90.79</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">83.67</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">70.58/89.82</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">85.72</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">75.26</span>\n",
    "            </td>\n",
    "        </tr>\n",
    "        <tr>\n",
    "            <td rowspan=1 align=center> 20L1024H </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\"><b>ERNIE 3.0-Xbase-zh</b></span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\"><b>78.39</b></span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\"><b>76.16</b></span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\"><b>59.55</b></span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\"><b>61.87</b></span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\"><b>84.40</b></span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\"><b>81.73</b></span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\"><b>88.82</b></span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\"><b>83.60</b></span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\"><b>75.99/93.00</b></span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\"><b>86.78</b></span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\"><b>84.98</b></span>\n",
    "            </td>\n",
    "        </tr>\n",
    "        <tr>\n",
    "            <td rowspan=9 align=center> 12L768H </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">\n",
    "                    <a href=\"https://bj.bcebos.com/paddlenlp/models/transformers/ernie_3.0/ernie_3.0_base_zh.pdparams\">\n",
    "                        ERNIE 3.0-Base-zh\n",
    "                    </a>\n",
    "                </span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">76.05</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">75.93</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">58.26</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">61.56</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">83.02</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\"><b>80.10</b></span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">86.18</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">82.63</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">70.71/90.41</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">84.26</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\"><b>77.88</b></span>\n",
    "            </td>\n",
    "        </tr>\n",
    "        <tr>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">ERNIE 1.0-Base-zh-cw</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\"><b>76.47</b></span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\"><b>76.07</b></span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">57.86</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">59.91</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\"><b>83.41</b></span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">79.58</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\"><b>89.91</b></span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\"><b>83.42</b></span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\"><b>72.88/90.78</b></span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\"><b>84.68</b></span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">76.98</span>\n",
    "            </td>\n",
    "        </tr>\n",
    "        <tr>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">ERNIE-Gram-zh</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">75.72</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">75.28</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">57.88</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">60.87</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">82.90</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">79.08</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">88.82</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">82.83</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">71.82/90.38</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">84.04</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">73.69</span>\n",
    "            </td>\n",
    "        </tr>\n",
    "        <tr>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">Langboat/Mengzi-BERT-Base</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">74.69</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">75.35</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">57.76</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">61.64</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">82.41</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">77.93</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">88.16</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">82.20</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">67.04/88.35</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">83.74</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">70.70</span>\n",
    "            </td>\n",
    "        </tr>\n",
    "        <tr>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">ERNIE 2.0-Base-zh</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">74.32</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">75.65</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">58.25</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">61.64</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">82.62</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">78.71</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">81.91</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">82.33</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">66.08/87.46</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">82.78</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">73.19</span>\n",
    "            </td>\n",
    "        </tr>\n",
    "        <tr>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">ERNIE 1.0-Base-zh</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">74.17</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">74.84</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\"><b>58.91</b></span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\"><b>62.25</b></span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">81.68</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">76.58</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">85.20</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">82.77</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">67.32/87.83</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">82.47</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">69.68</span>\n",
    "            </td>\n",
    "        </tr>\n",
    "        <tr>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">RoBERTa-wwm-ext</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">74.11</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">74.60</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">58.08</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">61.23</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">81.11</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">76.92</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">88.49</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">80.77</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">68.39/88.50</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">83.43</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">68.03</span>\n",
    "            </td>\n",
    "        </tr>\n",
    "        <tr>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">BERT-Base-Chinese</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">72.57</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">74.63</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">57.13</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">61.29</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">80.97</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">75.22</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">81.91</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">81.90</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">65.30/86.53</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">82.01</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">65.38</span>\n",
    "            </td>\n",
    "        </tr>\n",
    "        <tr>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">UER/Chinese-RoBERTa-Base</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">71.78</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">72.89</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">57.62</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">61.14</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">80.01</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">75.56</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">81.58</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">80.80</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">63.87/84.95</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">81.52</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">62.76</span>\n",
    "            </td>\n",
    "        </tr>\n",
    "        <tr>\n",
    "            <td rowspan=1 align=center> 8L512H </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">UER/Chinese-RoBERTa-Medium</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">67.06</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">70.64</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">56.10</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">58.29</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">77.35</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">71.90</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">68.09</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">78.63</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">57.63/78.91</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">75.13</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">56.84</span>\n",
    "            </td>\n",
    "        </tr>\n",
    "        <tr>\n",
    "            <td rowspan=5 align=center> 6L768H </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">\n",
    "                    <a href=\"https://bj.bcebos.com/paddlenlp/models/transformers/ernie_3.0/ernie_3.0_medium_zh.pdparams\">\n",
    "                        ERNIE 3.0-Medium-zh\n",
    "                    </a>\n",
    "                </span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\"><b>72.49</b></span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\"><b>73.37</b></span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\"><b>57.00</b></span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">60.67</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\"><b>80.64</b></span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\"><b>76.88</b></span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\"><b>79.28</b></span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\"><b>81.60</b></span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\"><b>65.83/87.30</b></span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\"><b>79.91</b></span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\"><b>69.73</b></span>\n",
    "            </td>\n",
    "        </tr>\n",
    "        <tr>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">HLF/RBT6, Chinese</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">70.06</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">73.45</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">56.82</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">59.64</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">79.36</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">73.32</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">76.64</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">80.67</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">62.72/84.77</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">78.17</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">59.85</span>\n",
    "            </td>\n",
    "        </tr>\n",
    "        <tr>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">TinyBERT<sub>6</sub>, Chinese</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">69.62</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">72.22</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">55.70</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">54.48</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">79.12</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">74.07</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">77.63</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">80.17</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">63.03/83.75</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">77.64</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">62.11</span>\n",
    "            </td>\n",
    "        </tr>\n",
    "        <tr>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">RoFormerV2 Small</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">68.52</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">72.47</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">56.53</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\"><b>60.72</b></span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">76.37</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">72.95</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">75.00</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">81.07</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">62.97/83.64</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">67.66</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">59.41</span>\n",
    "            </td>\n",
    "        </tr>\n",
    "        <tr>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">UER/Chinese-RoBERTa-L6-H768</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">67.09</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">70.13</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">56.54</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">60.48</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">77.49</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">72.00</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">72.04</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">77.33</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">53.74/75.52</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">76.73</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">54.40</span>\n",
    "            </td>\n",
    "        </tr>\n",
    "        <tr>\n",
    "            <td rowspan=1 align=center> 6L384H </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">\n",
    "                    <a href=\"https://bj.bcebos.com/paddlenlp/models/transformers/ernie_3.0/ernie_3.0_mini_zh.pdparams\">\n",
    "                        ERNIE 3.0-Mini-zh\n",
    "                    </a>\n",
    "                </span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">66.90</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">71.85</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">55.24</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">54.48</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">77.19</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">73.08</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">71.05</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">79.30</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">58.53/81.97</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">69.71</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">58.60</span>\n",
    "            </td>\n",
    "        </tr>\n",
    "               <tr>\n",
    "            <td rowspan=1 align=center> 4L768H </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">HFL/RBT4, Chinese</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">67.42</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">72.41</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">56.50</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">58.95</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">77.34</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">70.78</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">71.05</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">78.23</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">59.30/81.93</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">73.18</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">56.45</span>\n",
    "            </td>\n",
    "        </tr>\n",
    "        <tr>\n",
    "            <td rowspan=1 align=center> 4L512H </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">UER/Chinese-RoBERTa-Small</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">63.25</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">69.21</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">55.41</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">57.552</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">73.64</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">69.80</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">66.78</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">74.83</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">46.75/69.69</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">67.59</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">50.92</span>\n",
    "            </td>\n",
    "        </tr>\n",
    "        <tr>\n",
    "            <td rowspan=1 align=center> 4L384H </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">\n",
    "                    <a href=\"https://bj.bcebos.com/paddlenlp/models/transformers/ernie_3.0/ernie_3.0_micro_zh.pdparams\">\n",
    "                    ERNIE 3.0-Micro-zh\n",
    "                    </a>\n",
    "                </span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">64.21</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">71.15</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">55.05</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">53.83</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">74.81</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">70.41</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">69.08</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">76.50</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">53.77/77.82</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">62.26</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">55.53</span>\n",
    "            </td>\n",
    "        </tr>\n",
    "        <tr>\n",
    "            <td rowspan=2 align=center> 4L312H </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">\n",
    "                    <a href=\"https://bj.bcebos.com/paddlenlp/models/transformers/ernie_3.0/ernie_3.0_nano_zh.pdparams\">\n",
    "                        ERNIE 3.0-Nano-zh\n",
    "                    </a>\n",
    "                </span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\"><b>62.97</b></span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\"><b>70.51</b></span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\"><b>54.57</b></span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\"><b>48.36</b></span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\"><b>74.97</b></span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\"><b>70.61</b></span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">68.75</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\"><b>75.93</b></span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\"><b>52.00/76.35</b></span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\"><b>58.91</b></span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\"><b>55.11</b></span>\n",
    "            </td>\n",
    "        </tr>\n",
    "        <tr>\n",
    "            <td style=\"text-align:center\">\n",
    "            <span style=\"font-size:18px\">TinyBERT<sub>4</sub>, Chinese</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">60.82</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">69.07</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">54.02</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">39.71</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">73.94</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">69.59</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\"><b>70.07</b></span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">75.07</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">46.04/69.34</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">58.53</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">52.18</span>\n",
    "            </td>\n",
    "        </tr>\n",
    "        <tr>\n",
    "            <td rowspan=1 align=center> 4L256H </td>\n",
    "            <td style=\"text-align:center\">\n",
    "            <span style=\"font-size:18px\">UER/Chinese-RoBERTa-Mini</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">53.40</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">69.32</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">54.22</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">41.63</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">69.40</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">67.36</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">65.13</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">70.07</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">5.96/17.13</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">51.19</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">39.68</span>\n",
    "            </td>\n",
    "        </tr>\n",
    "        <tr>\n",
    "            <td rowspan=1 align=center> 3L1024H </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">HFL/RBTL3, Chinese</span>\n",
    "            </td>\n",
    "                <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">66.63</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">71.11</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">56.14</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">59.56</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">76.41</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">71.29</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">69.74</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">76.93</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">58.50/80.90</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">71.03</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">55.56</span>\n",
    "            </td>\n",
    "        </tr>\n",
    "        <tr>\n",
    "            <td rowspan=1 align=center> 3L768H </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">HFL/RBT3, Chinese</span>\n",
    "            </td>\n",
    "                <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">65.72</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">70.95</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">55.53</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">59.18</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">76.20</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">70.71</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">67.11</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">76.63</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">55.73/78.63</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">70.26</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">54.93</span>\n",
    "            </td>\n",
    "        </tr>\n",
    "        <tr>\n",
    "            <td rowspan=1 align=center> 2L128H </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">UER/Chinese-RoBERTa-Tiny</span>\n",
    "            </td>\n",
    "                <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">44.45</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">69.02</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">51.47</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">20.28</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">59.95</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">57.73</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">63.82</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">67.43</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">3.08/14.33</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">23.57</span>\n",
    "            </td>\n",
    "            <td style=\"text-align:center\">\n",
    "                <span style=\"font-size:18px\">28.12</span>\n",
    "            </td>\n",
    "        </tr>\n",
    "    <tbody>\n",
    "</table>\n",
    "<br />\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 3.模型如何使用\n",
    "\n",
    "安装 paddlenlp 最新的安装包\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-11-11T06:50:18.329409Z",
     "iopub.status.busy": "2022-11-11T06:50:18.328968Z",
     "iopub.status.idle": "2022-11-11T06:53:07.814814Z",
     "shell.execute_reply": "2022-11-11T06:53:07.813775Z",
     "shell.execute_reply.started": "2022-11-11T06:50:18.329379Z"
    },
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "!pip install paddlenlp --upgrade"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 微调\n",
    "\n",
    "使用 PaddleNLP 只需要一行代码可以拿到 ERNIE 3.0 系列预训练模型，之后可以在自己的下游数据下进行微调，从而获得具体任务上效果更好的模型。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-11-11T06:53:07.817337Z",
     "iopub.status.busy": "2022-11-11T06:53:07.816843Z",
     "iopub.status.idle": "2022-11-11T06:53:56.003766Z",
     "shell.execute_reply": "2022-11-11T06:53:56.002942Z",
     "shell.execute_reply.started": "2022-11-11T06:53:07.817293Z"
    },
    "scrolled": true,
    "tags": []
   },
   "outputs": [],
   "source": [
    "from paddlenlp.transformers import *\n",
    "\n",
    "tokenizer = AutoTokenizer.from_pretrained(\"ernie-3.0-medium-zh\")\n",
    "\n",
    "# 用于分类任务\n",
    "seq_cls_model = AutoModelForSequenceClassification.from_pretrained(\"ernie-3.0-medium-zh\")\n",
    "\n",
    "# 用于序列标注任务\n",
    "token_cls_model = AutoModelForTokenClassification.from_pretrained(\"ernie-3.0-medium-zh\")\n",
    "\n",
    "# 用于阅读理解任务\n",
    "qa_model = AutoModelForQuestionAnswering.from_pretrained(\"ernie-3.0-medium-zh\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "可启动如下微调脚本对 **ERNIE 3.0-Medium** 在 CLUE IFLYTEK 文本分类的数据集上进行微调：\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-11-11T06:53:56.006231Z",
     "iopub.status.busy": "2022-11-11T06:53:56.005136Z",
     "iopub.status.idle": "2022-11-11T06:55:21.095442Z",
     "shell.execute_reply": "2022-11-11T06:55:21.094490Z",
     "shell.execute_reply.started": "2022-11-11T06:53:56.006195Z"
    },
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "!git clone https://gitee.com/paddlepaddle/PaddleNLP.git"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-11-11T06:56:23.255848Z",
     "iopub.status.busy": "2022-11-11T06:56:23.255438Z",
     "iopub.status.idle": "2022-11-11T06:58:21.333380Z",
     "shell.execute_reply": "2022-11-11T06:58:21.332474Z",
     "shell.execute_reply.started": "2022-11-11T06:56:23.255822Z"
    },
    "scrolled": true,
    "tags": []
   },
   "outputs": [],
   "source": [
    "# 分类任务\n",
    "# 该脚本共支持 CLUE 中 7 个分类任务，超参不全相同，因此分类任务中的超参配置利用 config.yml 配置\n",
    "!python PaddleNLP/model_zoo/ernie-3.0/run_seq_cls.py  \\\n",
    "    --task_name iflytek \\\n",
    "    --model_name_or_path ernie-3.0-medium-zh \\\n",
    "    --do_train\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 模型压缩\n",
    "\n",
    "如果有模型部署上线的需求，则可以进一步压缩模型体积，可使用模型压缩方案及 API 对上一步微调后的模型进行压缩。\n",
    "\n",
    "模型压缩 API 的使用可参考[文档](../../docs/compression.md)。同样地，模型压缩 API 也支持分类（包含文本分类、文本匹配、自然语言推理、代词消歧等任务）、序列标注、阅读理解、信息抽取等自然语言处理场景。\n",
    "\n",
    "压缩后保存的模型可以直接用于部署。\n",
    "\n",
    "## 部署\n",
    "\n",
    "我们为 ERNIE 3.0 提供了[多种部署方案](https://github.com/paddlepaddle/PaddleNLP/tree/develop/model_zoo/ernie-3.0#%E9%83%A8%E7%BD%B2)，可以满足不同场景下的部署需求，请根据实际情况进行选择：\n",
    "<p align=\"center\">\n",
    "        <img width=\"700\" alt=\"image\" src=\"https://user-images.githubusercontent.com/26483581/175260618-610a160c-270c-469a-842c-96871243c4ed.png\">\n",
    "</p>\n",
    "\n",
    "\n",
    "# 4.原理\n",
    "\n",
    "### 在线蒸馏技术\n",
    "\n",
    "在线蒸馏技术在模型学习的过程中周期性地将知识信号传递给若干个学生模型同时训练，从而在蒸馏阶段一次性产出多种尺寸的学生模型。相对传统蒸馏技术，该技术极大节省了因大模型额外蒸馏计算以及多个学生的重复知识传递带来的算力消耗。\n",
    "\n",
    "这种新颖的蒸馏方式利用了文心大模型的规模优势，在蒸馏完成后保证了学生模型的效果和尺寸丰富性，方便不同性能需求的应用场景使用。此外，由于文心大模型的模型尺寸与学生模型差距巨大，模型蒸馏难度极大甚至容易失效。为此，通过引入了助教模型进行蒸馏的技术，利用助教作为知识传递的桥梁以缩短学生模型和大模型表达空间相距过大的问题，从而促进蒸馏效率的提升。\n",
    "\n",
    "更多技术细节可以参考论文：\n",
    "- [ERNIE-Tiny: A Progressive Distillation Framework for Pretrained Transformer Compression](https://arxiv.org/abs/2106.02241)\n",
    "- [ERNIE 3.0 Titan: Exploring Larger-scale Knowledge Enhanced Pre-training for Language Understanding and Generation](https://arxiv.org/abs/2112.12731)\n",
    "\n",
    "<p align=\"center\">\n",
    "        <img width=\"644\" alt=\"image\" src=\"https://user-images.githubusercontent.com/1371212/168516904-3fff73e0-010d-4bef-adc1-4d7c97a9c6ff.png\" title=\"ERNIE 3.0 Online Distillation\">\n",
    "</p>\n",
    "\n",
    "\n",
    "\n",
    "# 5.相关论文及引用信息\n",
    "\n",
    "\n",
    "```text\n",
    "@article{sun2021ernie,\n",
    "  title={Ernie 3.0: Large-scale knowledge enhanced pre-training for language understanding and generation},\n",
    "  author={Sun, Yu and Wang, Shuohuan and Feng, Shikun and Ding, Siyu and Pang, Chao and Shang, Junyuan and Liu, Jiaxiang and Chen, Xuyi and Zhao, Yanbin and Lu, Yuxiang and others},\n",
    "  journal={arXiv preprint arXiv:2107.02137},\n",
    "  year={2021}\n",
    "}\n",
    "\n",
    "@article{su2021ernie,\n",
    "  title={Ernie-tiny: A progressive distillation framework for pretrained transformer compression},\n",
    "  author={Su, Weiyue and Chen, Xuyi and Feng, Shikun and Liu, Jiaxiang and Liu, Weixin and Sun, Yu and Tian, Hao and Wu, Hua and Wang, Haifeng},\n",
    "  journal={arXiv preprint arXiv:2106.02241},\n",
    "  year={2021}\n",
    "}\n",
    "\n",
    "@article{wang2021ernie,\n",
    "  title={Ernie 3.0 titan: Exploring larger-scale knowledge enhanced pre-training for language understanding and generation},\n",
    "  author={Wang, Shuohuan and Sun, Yu and Xiang, Yang and Wu, Zhihua and Ding, Siyu and Gong, Weibao and Feng, Shikun and Shang, Junyuan and Zhao, Yanbin and Pang, Chao and others},\n",
    "  journal={arXiv preprint arXiv:2112.12731},\n",
    "  year={2021}\n",
    "}\n",
    "```\n",
    "\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "py35-paddle1.2.0"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
